Finding Image Regions with Human Computation and Games with a Purpose

Manual image annotation is a tedious and timeconsuming task, while automated methods are error prone and limited in their results. Human computation, and especially games with a purpose, have shown potential to create high quality annotations by “hiding the complexity” of the actual annotation task and employing the “wisdom of the crowds”. In this demo paper we present two games with a single purpose: finding regions in images that correspond to given terms. We discuss approach, implementation, and preliminary results of our work and give an outlook to immediate future work. Automatic image understanding is limited. In image and visual information retrieval the main problem is defined as the semantic gap, which is the gap between low level features, like color, texture and shapes, and high level understanding like image semantics. Not only the image as a whole cannot be interpreted solely by algorithms, but also relation between image regions and semantics cannot be found in a fully automated way. Many applications would profit from an algorithm determining which pixels of an image correspond to which semantic concepts. Figure 1 gives an example photo on the left side. The photo (taken from flickr.com, from one of the authors’ photo stream) is tagged squirrel. However, the semantic concept squirrel is covered by the small region on the right hand side, a fraction of the original image. With this information algorithms for saliency maps on images, e.g. (Itti, Koch, and Niebur 1998), image re-targeting, e.g. (Avidan and Shamir 2007), etc. could be leveraged. Note at this point that the region heavily depends on the concept given by the terms. Annotations like squirrel on a handrail or squirrel in a park need significantly larger regions in our example in figure 1. Automated localization of concepts has limited success. (Lampert, Blaschko, and Hofmann 2008) for instance report average precision values of 0.223 and 0.148 for two selected concepts on the PASCAL VOC 2006 data set. So besides the limitation in precision and recall there is also a limitation in the number of concepts to be localized. (Liu et al. 2011) report higher precision and recall values for finding salient Copyright c © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. regions in images, but while this is a generic approach for a broad range of objects and concepts, it’s based on salient objects and does not assign labels to salient regions found. Therefore, finding regions corresponding to terms with images human computation (von Ahn 2007) is a valid alternative to automated approaches. Especially games with a purpose (von Ahn 2006), which are fun, simple and entertaining games that hide the actual task and motivate users on a playful level to do things they otherwise wouldn’t do, have a huge potential. Image annotation & games with a purpose are quite a prominent combination. Labeling images in a competitive multiplayer game has been one of the very first applications called games with a purpose, cp. The ESP Game (von Ahn and Dabbish 2004). An effort very similar to our approach is PeekaBoom (von Ahn, Liu, and Blum 2006). In this game two players simultaneously play a cooperative 2-player game, where one player reveals regions of an image according to given terms, while the other guesses terms that apply to the uncovered region. Score points earned depend on how few pixels had to be revealed for the correct term to be guessed. Motivation for the users is to enter a global highscore. Another related effort is LabelMe (Russell et al. 2008), a web based annotation tool, where users can annotate images by drawing regions and assigning text to them. However, LabelMe is not a game, but a collaborative annotation tool. Figure 1: Example photo with the original version on the left hand side and the region covering the semantic concept squirrel on the right hand side. In this publication we present two games with a purpose, called RecognizePicture and rpMobile, trying to detect regions that cover the semantics of the given tags. These are not necessarily objects in the images, but also concepts like car race, summer or beautiful. Human Computation and Serious Games: Papers from the 2012 AIIDE Joint Workshop AAAI Technical Report WS-12-17